Processing for accurate reproduction of symbols and other high-frequency areas in a color image

ABSTRACT

In image processing for accurate reproduction of the color of a high-frequency area, a histogram is calculated that is consonant with an input image, and, based on the histogram, there is calculated a binary threshold value with which a predetermined area in the input image is blurred. The input image is binarized using the binary threshold value, and the color of the predetermined area of the input image is calculated based on the results obtained in the binarization.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, and a program and a storage medium therefor.

2. Related Background Art

Recently, as a result of the current widespread employment of scanners,the digitization of documents has become a popular practice. However,for the storage, in bit map form, of a full-color, A4-sized digitaldocument that has been scanned at 300 dpi, for example, a huge amount ofmemory, upwards of 24 Mbytes, must be allocated. And such a large datarecording is not amenable to being attached to and transmitted withmail.

Therefore, JPEG, a well known compression technique, is commonly used tocompress full-color image data. With JPEG, however, although it is veryeffective when used to compress natural images, such as photographs, andthe quality of the images produced when it is used is high, whenhigh-frequency portions, such as symbols, are compressed using JPEG,image deterioration called mosquito noise occurs, and the compressionrate is also reduced. Therefore, since generally an office documentincludes many symbol portions, after a document is binarized, MMR isused to compress the binary document and obtain the coordinates ofsymbol portions and the representative colors of the symbols therein, sothat an office document prepared in color can be easily represented.Further, for a complicated color document, such as a magazine, an areato be compressed is divided into background and symbol portions, andwhile the background is compressed using JPEG, symbols are binarizedusing an optimal threshold value and the obtained binary images arecompressed using MMR, following which color information is added to theobtained MMR data. In this manner, even a fairly complicated colordocument can be represented using a small data file.

Therefore, a technique is required for calculating the representativecolor of symbols in a symbol portion. The following is an exampleconventional method used for calculating the representative symbolcolor.

First, a rough, three-dimensional histogram is prepared for multi-valuedimage data in a black portion by referring to the binary image of asymbol area. Then, a fine histogram is prepared for the pixels of amulti-valued image that corresponds to the highest value in the roughthree-dimensional histogram, and the highest value that is therebyobtained is determined to be the representative color.

However, when the above method is employed to calculate therepresentative color of symbol colors, although a desirable color can becalculated for a symbol having a height of 12 points or more when readat a resolution of 300 dpi or higher, for a 10 point or smaller symbol,the ratio of the originally calculated representative color data to theblack of the binary image is small, and a desired color can not becalculated.

An explanation will now be given, while referring to FIG. 19, for a casewherein calculations are performed to obtain the representative color ofa large symbol, and for a case wherein calculation are performed toobtain the representative color of a small symbol.

FIG. 19 is a diagram showing a sample wherein green symbols are writtenon a white background. A binary result 1901 is obtained for acomparatively thick symbol, and the multi-valued image of a blackportion in the binary result 1901 has a level change 1902. In the levelchange 1902, since the level remains steady for a long time at portions1903 and 1904, which correspond to the representative color of thesymbol, the color is distributed in the color space RGB as is shown inFIG. 20A. A block 2002 in FIG. 20A is the color green in FIG. 19, i.e.,indicates the representative color of the symbol. Since the block 2002,of the symbol portion, has a specific size, it can be extractedcomparatively easily.

But then, for a fine symbol 1906 in FIG. 19, the level change in amulti-valued image has a shape 1907, and as soon as the level reachesportions 1908 and 1909, which correspond to the representative color ofthe symbol, it is changed to the level of the background portion. Inthis case, the color distribution in the RGB color space is as is shownin FIG. 20B, and compared with the block 2002 in FIG. 20A, using theobtained data it is difficult to calculate a point 2005 in FIG. 20B.Through the binarization process, the left side of a broken line isbinarized as a black symbol, and when the representative color iscalculated using the conventional method, the point 2005 is obtained asa value, the greatest number that is present. This is not preferablebecause compared with the desired symbol color, the obtained symbol hasa whitish-green cast.

In order to avoid the occurrence of this phenomenon, a method isavailable whereby a binary image is thinned and a conventionalrepresentative calculation is performed using a fine image. When thismethod is applied, however, a defect described in the followingexplanation occurs.

To simplify the explanation, a symbol “∘” is used as an example.

Assume that in FIG. 21 a green symbol “∘” is drawn on a whitebackground. The level shift for the symbol “∘” has a change 2104.Originally it would be ideal for the center indentation to be returnedto the white level; however, the complete return to the white level ofthe symbol “∘”, a small point, may not be possible. If the binarizationprocess is performed by using a threshold value 2105, a solid black dot2102 is obtained as the binary result. And if the thinning process isthen performed for this dot 2102, a black dot 2103 is obtained. Inaccordance with the level 2104, the position of the multi-valued imageindicated by this binary image is a point 2106, which is not apreferable level for the representative color.

Since this “crushed phenomenon” occurs for a symbol having a smallpoint, it is apparent that the thinning process is not effective.

The binary image that is the output employed for representing a symbolis used to calculate the representative color for the symbol. However,it is preferable that a threshold value for optimally representing asymbol be binarized, so that no blurring of the symbol occurs. It isfurther known that, while taking the succeeding OCR process intoaccount, it is better for a binarized symbol to become solid than it isfor it to become blurred, since better OCR results can be obtained.

FIG. 22 is a graph showing a typical histogram for the brightness of asymbol area. A point 2201 is a desirable point for a binary image.However, when binarization is performed at this point 2201, a pixel thatis shifted from the background to the symbol portion is binarized as ablack dot, a preferable output, while when the calculation of therepresentative color of the symbol is performed, this output constitutesnoise.

This state is shown in FIG. 22. When binarization is performed at thepoint 2201 in FIG. 22, this is the equivalent of binarization beingperformed at a level 2301 in FIG. 23, and the binary image that isobtained also includes many portions 2302 and 2303 that are shifted fromthe background to the symbol.

As is described above, since the binary image that is the outputemployed to represent a symbol is used for the calculation of therepresentative color of the symbol, it is not possible to calculate anoptimal representative color for the symbol portion.

Furthermore, according to the conventional method, for each symbol areaonly one representative color can be obtained, and a symbol area inwhich multiple colors appear can not be coped with.

SUMMARY OF THE INVENTION

To resolve the above problems, it is one objective of the presentinvention to provide an image processing apparatus and an imageprocessing method for calculating the optimal representative color for asymbol portion, and to provide a storage medium therefor.

To achieve this objective, according to the present invention, an imageprocessing apparatus comprises:

histogram calculation means for calculating a histogram that isconsonant with an input image;

binary threshold value calculation means for calculating a binarythreshold value, based on the histogram, with which a predetermined areain the input image is blurred;

binarization means for binarizing the input image using the binarythreshold value; and

calculation means for calculating the color of the predetermined area ofthe input image based on the results obtained by the binarization means.

Further, to resolve the above described problems, it is anotherobjective of the present invention to provide an image processingapparatus and an image processing method for allocating multiple colorsto a symbol area, and a storage medium therefor.

To achieve this objective, according to the present invention, an imageprocessing apparatus comprises:

binarization means for binarizing color image data;

detection means for detecting a symbol area in the color image data;

color reduction means for introducing, from N colors that constitute asymbol in the symbol area, M colors that are equal to or smaller thanthe N colors;

symbol cutting means for performing a symbol cutting process for thesymbol area; and

color allocation means for allocating one of the M colors for each cutsymbol unit obtained by the symbol cutting means.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing the configuration of an image processingapparatus according to a first embodiment of the present invention;

FIG. 2 is a diagram showing an example original image;

FIG. 3 is a flowchart showing the processing performed by a binarizingunit and an area division unit;

FIG. 4 is a graph showing a brightness histogram for the original image;

FIG. 5 is a diagram showing a binary image obtained by binarizing theoriginal image;

FIG. 6 is a diagram showing the state wherein label attachment isperformed for the binary image;

FIG. 7 is a diagram showing black areas that are extracted from theoriginal image in accordance with a symbol attribute;

FIG. 8 is a diagram showing symbol areas in the original image;

FIG. 9 is a flowchart showing the processing performed by a symbol,representative color operation unit;

FIG. 10 is a flowchart showing an example method employed forcalculating a threshold value T2;

FIG. 11 is a diagram showing the configuration of an image processingapparatus according to a second embodiment of the present invention;

FIG. 12 is a diagram showing a difference between normal symbols andinverted symbols;

FIG. 13 is a graph showing a brightness histogram for the invertedsymbol portion;

FIG. 14 is a diagram showing the configuration of an image processingapparatus according to a third embodiment of the present invention;

FIG. 15 is a diagram showing the configuration according to the thirdembodiment for expanding compressed data;

FIG. 16 is a flowchart showing the processing performed for the invertedsymbol;

FIG. 17 is a diagram showing an example histogram shape;

FIG. 18 is a diagram showing an example method used for calculating athreshold value T2;

FIG. 19 is a diagram used for explaining a case wherein therepresentative color for a large symbol is calculated and a case whereinthe representative color for a small symbol is calculated;

FIGS. 20A and 20B are diagrams showing distributions in the RGB space;

FIG. 21 is a diagram for explaining the thinning of a binary image;

FIG. 22 is a graph showing a typical brightness for a symbol area;

FIG. 23 is a diagram showing the results obtained by binarizing animage;

FIG. 24 is a diagram showing the results obtained by binarizing animage;

FIG. 25 is a diagram showing the configuration of an image processingapparatus according to a fourth embodiment of the present invention;

FIG. 26 is a diagram showing the arrangement of the image processingapparatus for expanding compressed data according to the fourthembodiment;

FIG. 27 is a flowchart showing the symbol area detection processingperformed according to the fourth embodiment of the present invention;

FIG. 28 is a diagram for explaining the symbol area detection processingperformed according to the fourth embodiment of the present invention;

FIG. 29 is a diagram for explaining the symbol area detection processingperformed according to the fourth embodiment of the present invention;

FIG. 30 is a diagram for explaining the symbol area detection processingperformed according to the fourth embodiment of the present invention;

FIG. 31 is a diagram for explaining the symbol area detection processingperformed according to the fourth embodiment of the present invention;

FIG. 32 is a diagram for explaining the symbol area detection processingperformed according to the fourth embodiment of the present invention;

FIG. 33 is a diagram for explaining the processing performed for there-binarizing a symbol area according to the fourth embodiment of theinvention;

FIGS. 34A, 34B and 34C are diagrams for explaining the symbol paintingprocessing performed according to the fourth embodiment of the presentinvention;

FIG. 35 is a flowchart for explaining the symbol painting processingperformed according to the fourth embodiment of the present invention;

FIG. 36 is a flowchart for explaining the one color extractionprocessing performed according to the fourth embodiment of the presentinvention;

FIG. 37 is a diagram for explaining the one color extraction processingperformed according to the fourth embodiment of the present invention;

FIGS. 38A, 38B and 38C are diagrams for explaining the state wherein theimage processing apparatus of the fourth embodiment expands compresseddata and combines the obtained data;

FIG. 39 is a diagram for explaining the color reduction processingperformed according to the fourth embodiment of the present invention;

FIG. 40 is a diagram showing the configuration of a modification of theimage processing apparatus of the fifth embodiment;

FIGS. 41A and 41B are diagrams for explaining the image compressionprocessing performed according to the modification;

FIGS. 42A, 42B and 42C are diagrams for explaining the processingperformed when binarizing a symbol area according to the modification;

FIG. 43 is a flowchart showing the symbol color extraction processingperformed according to the fifth embodiment of the present invention;

FIG. 44 is a flowchart showing the color reduction processing performedaccording to the fifth embodiment of the present invention;

FIG. 45 is a flowchart showing the color reduction processing performedaccording to the fifth embodiment of the present invention;

FIG. 46 is a diagram for explaining the shifting portion (gradation) ofa symbol that is generated by a scanner;

FIG. 47 is a diagram for explaining the color reduction processingperformed according to the fifth embodiment of the present invention;

FIG. 48 is a diagram for explaining the color reduction processingperformed according to the fifth embodiment of the present invention;

FIG. 49 is a diagram showing a three-dimensional histogram thatrepresents the shifting portion (gradation) of a symbol that isgenerated by a scanner; and

FIG. 50 is a diagram for explaining the color allocation processingperformed according to the fifth embodiment of the present invention fordetermining the color of each symbol using symbol cutting information.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following embodiments, symbol contains character, and so on.

(First Embodiment)

The preferred embodiments of the present invention will now be describedin detail while referring to the drawings.

FIG. 1 is a diagram showing the configuration of an image processingapparatus according to this embodiment.

A binarizing unit (a) 102 binarizes an input original image 101, andgenerates a binary image (a) 103.

An area division unit 104 detects a symbol area or a photograph area inthe received binary image (a) 103, and generates the coordinates of thearea and an attribute, such as area information 105, for example, for asymbol or a photograph.

Based on the area information 105, an MMR compression unit 106 performsMMR compression for a part of the binary image (a) 103 that correspondsto the area having a symbol attribute, and generates compressed code D107.

Based on the area information 105, a symbol representative coloroperation unit 108 calculates the representative color of a symbol thatis included in the area that corresponds to the area having the symbolattribute. A binarizing unit (b) 1081, for calculating the symbolrepresentative color, is included in the symbol representative coloroperation unit 108 and generates a binary image (b) 1082. The colorinformation obtained during this process is newly written as theattribute of the area information 105.

Based on the area information 105, a JPEG compression unit 109compresses a part of the original image, which is included in the areacorresponding to the area including the attribute of a natural image,and generates compressed code C 110.

This configuration will now be described in more detail.

FIG. 3 is a flowchart showing the processing performed by the binarizingunit (a) 102 and the area division unit 104.

Steps S301 to S303 show the processing performed by the binarizing unit(a) 102, and steps S304 to S306 show the processing performed by thearea division unit 104.

At step S301, an original image 101, such as an RGB color image, isinput, and the brightness conversion for this image is performed byusing the following equation to generate a brightness image J.Y=0.299R+0.587G+0.114B.

At step S302 the brightness data histogram is prepared, and thethreshold value T used for binarization is calculated.

At step S303, the brightness image J is binarized by using the thresholdvalue T, and a binary image K is generated.

At step S304, the borderline of a black pixel is traced, and a labelattachment is performed for each of the black areas.

At step S305, the forms and positions of the-black areas to which thelabels are attached are employed to determine whether the image is asymbol or a natural image.

At step S306, the symbol areas are combined in accordance with theirforms and positions, although the combining process at step S306 neednot always be performed. In this case, the number of symbol areas forwhich the representative color is calculated is increased and theprocessing time is extended, while the advantage is that a change in acolor can be coped with accurately.

An explanation will now be given for a case wherein the processing iscompleted up to the combining process at step S306 by using the originalimage in FIG. 2.

The brightness conversion is performed for the original image in FIG. 2(steps S301 and S302), and the obtained brightness histogram is as shownin FIG. 4. From this histogram, average and dispersion data are employedto calculate a threshold value T=150, and the obtained binary image, thebinary image 103 in FIG. 1, is as is shown in FIG. 5 (S303). FIG. 6 is adiagram showing the state wherein the border line tracing is performedfor the binary image in FIG. 5 at a reduced resolution, and labelattachment is performed for all the black areas (S304). The form andposition information of the black areas to which the labels are attachedis employed to determine the attribute of a symbol or a natural image(S305). It should be noted that this image is not actually generated butis merely a concept. In this example, since a portion 601 is large andcontains a black area, it is determined to be a natural image. Further,since areas 602 to 605 include symbols and have empty shapes, theirareas are determined to be frames. In this embodiment, frame informationis not included as area information 105, and is ignored. However, anapplication may hold the frame information, or may employ the frameinformation as the background for the symbol area information. In thiscase, means for calculating the color of the background must beprovided.

FIG. 7 is a diagram showing a black area wherein the symbol attribute isextracted from the original image in FIG. 2. When black pixels aregrouped in accordance with whether they are positioned near each otherand whether their widths and heights match, 17 symbol areas, 801 to 817,shown in FIG. 8 can be detected, as needed. In this embodiment, grouping(S306) is performed, and the 17 coordinate data sets, for which thesymbol attribute applies, are stored in the JPEG compression unit 109 inFIG. 1, while the coordinate data 601 in FIG. 6, for which a photographattribute applies, are stored in the JPEG compression unit 109 in FIG.1.

FIG. 9 is a flowchart showing the processing performed by the symbolrepresentative color operation unit 108. Since this processing isperformed for all the coordinates included in the area information 105,at step S901 a check is performed to determine whether there is aresymbol coordinates that have not been processed. If it is determinedthere are symbol coordinates that have not been processed, programcontrol advances to step S902, or if it is determined there are no suchcoordinates, the processing is terminated.

At step S902, a check is performed to determine whether a symbolattribute applies to the coordinates; if one does, program controladvances to step S903, whereas if one does not, program control returnsto step S901.

At step S903, a brightness histogram is calculated for the originalimage corresponding to the area information. Since this histogram is fora partial symbol area, it is highly probable that it does not have acomplicated shape comparable to that of the inclusive histogram shown inFIG. 4, but has a simple shape such as the one shown in FIG. 22.

At step S904, calculations are performed to obtain an optimal thresholdvalue, i.e., a threshold value T2 according to which the blurring of asymbol occurs, to be used for determining a representative color. Thisthreshold value T2 corresponds to the point 2202 in FIG. 22.

An example method for calculating the threshold value T2 will now bedescribed using the flowchart in FIG. 10.

At step S1001, a value of 0 is substituted into a variable “limit” forcounting the number of procedures, so that the processing does not enteran endless loop.

At step S1002, the brightness histogram is used for calculationsperformed to obtain an average value for the histogram and its skew,which is specially stored as skew_first. For these calculations, thefollowing equations are employed.${average} = {\sum\limits_{i = 0}^{255}\quad{{histgram}(i)}}$${skew\_ first} = {\sum\limits_{i = 0}^{255}\quad{( {i - {average}} )^{3}*{{histgram}(i)}}}$

At step S1003, “average” is substituted into HistUpper, and a value of 0is substituted into HistLower, following which, at step S1004, a checkis performed to determine whether a variable “limit” is equal to orgreater than 10. When the variable “limit” is equal to or greater than10, program control is shifted to step S1009 (in this case, instead of10, 5 or 20 may be employed). Then, at step S1005, HistUpper is used tocalculate the histogram for HistLower.${average} = {\sum\limits_{i = {HistLower}}^{HistUpper}\quad{{histgram}(i)}}$${myu} = {\sum\limits_{i = {HistLower}}^{HistUpper}\quad{( {i - {average}} )*{{histgram}(i)}}}$${skew} = {\sum\limits_{i = {HistLower}}^{HistUpper}\quad{( {i - {average}} )*{{histgram}(i)}}}$

At step S1006, a check is performed to determine whether the conditionskew <my*0.1 is satisfied, and when it is, no further calculations arerequired and program control jumps to step S1010. When this condition isnot satisfied, however, program control advances to step S1007 whereat acheck is performed to determine whether the conditions skew <0.0 andskew_first <skew*0.1 are satisfied. If these conditions are satisfied,no further calculations are required and program control jumps to stepS1010. But if these conditions are not satisfied, program controladvances to step S1008 and “average” is substituted into “HistLower”.Subsequently, at step S1009, the variable “limit” is incremented by oneand program control returns to step S1004.

By repeating this procedure, “average” is finally substituted into thethreshold value T2 at step S1010, and as a result, a threshold value isacquired that yields the blurred binary image indicated by the point2202 in FIG. 22.

Since the threshold value, according to which no black pixels arepresent when the image is binarized, could be obtained depending on theshape of a histogram, the number of pixels nearer the black area fromthe threshold value T2 is counted. When the number of pixels isextremely small, the pixels need to be more or less corrected so theyare closer to being white. The shape of the histogram that tends to echosuch results is shown in FIG. 17.

Instead of the complicated calculations described above, according toanother method for obtaining a histogram and for selecting a thresholdvalue, 5% of all pixels (this number is merely an example) are binarizedinto black dots. FIG. 18 is a diagram showing a threshold valuecalculation example.

At step S905, the binarizing unit (b) 1081 binarizes a partial areausing the threshold value T2, and generates the binary image (b) 1082 inFIG. 1. As is shown in FIG. 22, the area is binarized by using thethreshold value T2, i.e., the point 2202, and this means that the areais binarized at the level 2401 in FIG. 24, so that the binarization canbe performed without including the shifting portions 2402 and 2403.Then, as needed, the thinning process is performed for the obtainedbinary image. Since the threshold value is the one according to whichblurring of the binary image occurs, the probability is reduced that afailure will occur during the thinning process conventionally performedfor the representative color calculation, as explained while referringto FIG. 21. At step S906, a histogram is generated for each RGB pixel ofthe original image that corresponds to the black portions of the binaryimage (b). The color space for the histogram may not only be RGB, butmay also be YUV when the original image is YUV. At step S907, each ofthe peaks of the RGB histograms are defined as symbol representativecolors, and are written in the area information 105 as the attributes ofcorresponding areas.

The following other methods may be employed for steps S906 and S907. Forexample, instead of the histogram for each RGB pixel, a RGBthree-dimensional histogram is calculated. In this case, it isimpossible for the function of a calculator to calculate a histogram indetail, and it is preferable that a rough histogram be calculated inorder that it will not be affected by the noise produced by a colorshift point. As one method, first a highest value is obtained by usingthe rough histogram, then a detailed histogram present in the roughhistogram is calculated, and finally the highest value is obtainedagain.

Finally, according to the binary image area information, MMR compressionis performed for the area for which the symbol attribute applies andthat corresponds to the binary image (a) 102, and the compressed code D107 is generated. Also, JPEG compression is performed for the area forwhich the natural image attribute applies and that corresponds to theoriginal image data, and compressed code C 110 is generated. As needed,a format is generated by collecting the area information 105 thatincludes the area type, such as a symbol or a natural image, and therepresentative color when the image is a symbol, as well as thecompressed code C 111 and the compressed code D 112. The obtained formatis used as compressed data.

(Second Embodiment)

FIG. 11 is a diagram showing the configuration of an image processingapparatus according to a second embodiment of the present invention. Inthis embodiment, a binary image obtained by using a threshold value isnot employed as an image for which area division is performed. Instead,an edge amount relative to an adjacent pixel is calculated for all thepixels through differential filtering, and is binarized to obtain abinary image, and this binary image is used to perform area division.Border line tracing as used in the first embodiment is also used as anarea division method.

The difference between the first and this embodiment is that an areathat is to be extracted as a symbol also includes an area that would beinverted by normal binarization.

FIG. 12 is a diagram showing the difference between normal symbols andinverted symbols. An inverted symbol is, for example, a white symbol ona red background, not particularly a rarity in a color document. In thefirst embodiment, a symbol attribute is not provided for an invertedsymbol, and a natural nature attribute is provided for an area thatincludes an outer colored frame. In this embodiment, since adifferential binary image is used for area division, an inverted symbolarea, as shown in FIG. 12, can also be divided into symbols. In thiscase, the brightness histogram has the shape shown in FIG. 13, whilenormally a symbol area histogram has the shape shown in FIG. 22. A crestportion 1301 indicates a block of the background, and a crest portion1302 indicates a block of symbols. In this embodiment, the inversionprocess is required for a binarization process for generating a partialbinary image (b) (11082 in FIG. 11) used to calculate the symbolrepresentative color.

A determination as to whether a symbol is to be inverted can be madeusing the following equation.

The following equation is an example for the configuration in FIG. 1.${average} = {\sum\limits_{i = 0}^{255}\quad{{histgram}(i)}}$${skew\_ first} = {\sum\limits_{i = 0}^{255}\quad{( {i - {average}} )^{3}*{{histgram}(i)}}}$It can be ascertained that when skew_first is negative, the area is thenormal symbol portion shown in FIG. 22, and that when skew_first ispositive, the area is the inverted symbol portion shown in FIG. 13.

This processing will be briefly explained while referring to theflowchart in FIG. 16. In FIG. 16, the right side (S1613 to S1619) isexactly the same as the flowchart in FIG. 10, and the left side (S1605to S1611) is the calculation process for the inverted symbol.

When skew_first is greater than 0 at step S1603, a DoInvert flag, whichinstructs the binarization unit to perform an inversion process, is set.

When the DoInvert flag is set (ON), a binarizing unit (b) 11081 and abinarizing unit (a) 1111, which outputs the visual results, invert thebinarization results. In the arrangement provided to cope with aninverted symbol, an area division unit 1104 must detect an area having aframe attribute, and the average color in the frame must also becalculated. This is because the background color of the inverted symbolis other than white, and this color must be represented. The frame areaaverage color operation unit in charge of this process is not shown inFIG. 11.

With the arrangement in FIG. 14, which will be described later, sinceall the JPEG data is held for the background, an area having a frameattribute need not be prepared to cope with an inverted symbol.

(Third Embodiment)

The configuration of an image processing apparatus shown in FIG. 14 mayalso be employed.

The configuration in FIG. 14 will be briefly described.

In this configuration, a symbol area extraction unit 1402 for detectingonly the coordinates of a symbol area is provided for the area divisionprocess, and stores a symbol area coordinate 1403.

A binarizing unit 1404 generates a binary image 1405 of a symbol area,and in accordance with the binary image 1405, a symbol portion paintingunit 1408 generates a document 1413 wherein the symbol portion of theoriginal image is painted the average color of the surrounding portion.MMR compression is performed for the obtained partial binary image, andcompressed code D is generated, while JPEG compression is performed forthe symbol omission image and compressed code C is generated.

A symbol representative color operation unit 1411 performs theprocessing shown in FIG. 9 for the first embodiment, and generates arepresentative color 1412.

FIG. 15 is a diagram showing the configuration for expanding compresseddata obtained by the arrangement in FIG. 14.

To expand the compressed data, a JPEG expansion process is performed forthe compressed data C, and a multi-valued image G is generated. Further,the MMR expansion process is performed for the compressed code D, and abinary image F is generated for the partial area. Then, a combiningprocess is performed in which the representative value is added to theblack binary pixels in the image G while the binary white image isunchanged, and finally, an image H is obtained.

Compared with the configurations in FIGS. 14 and 15, the entire JPEGimage from which the symbol area is omitted is maintained, so that theatmosphere of the original image is not lost.

(Modification)

The present invention may be employed for a system that is constitutedby multiple apparatuses (e.g., a host computer, an interface device, areader or a printer), or for a single apparatus (e.g., a copier or afacsimile machine).

Further, the objective of the invention can also be achieved bysupplying, to a system or an apparatus (or a CPU or an MPU), a storagemedium (or a recording medium) on which software program code thatimplements the functions of the embodiments is recorded, and bypermitting the system or the apparatus to read and execute the recordedprogram code. In this case, the program code read from the storagemedium provides the functions of the above described embodiments, andthe storage medium on which the program code is recorded constitutes thepresent invention. In addition, with the present invention it is notonly possible for the functions of the previous embodiments to beprovided by the execution of program code by the computer, but also, theprogram code can interact with an Operating System (OS) running on acomputer, or with another software application, to provide the functionsdescribed in the above embodiments.

Furthermore, with the present invention, program code, read from astorage medium, can be written in a memory that is mounted on a functionexpansion board inserted into a computer, or in a function expansionunit connected to the computer, and in consonance with instructions inthe program code, a CPU mounted on the function expansion board, or inthe function expansion unit, can perform part or all of the actualprocessing required to implement the functions of the above describedembodiments.

As is described above, according to the present invention, a histogramconsonant with an input image is calculated, and is employed tocalculate the binary threshold value according to which a predeterminedarea in an image is blurred. The input image is binarized by using theobtained binary threshold value, and is employed to calculate the colorof the predetermined area of the input image. Therefore, even for a thinline symbol, the color data of the portion that is shifted from thebackground to the symbol can be deleted, so that an optimalrepresentative color can be obtained for the symbol.

(Fourth Embodiment)

An explanation will now be given for an image processing apparatusaccording to a fourth embodiment of the present invention thatefficiently compresses image data, while maintaining the informationconveyed by an original image, before storing a full color image on astorage medium or transmitting it via a transmission medium.

The image processing apparatus of this embodiment, first generates abrightness histogram for the entire image area, binarizes the imagearea, and extracts several symbol areas. Then, a symbol cutting processis performed for the individual symbol areas, and the results areemployed to determine whether each obtained area should be treated againas a symbol area. When the area should not be treated as a symbol area,a check is performed to determine whether an object in the pertinentarea has a single color. When the object has a single color, it isascertained that MMR compression should be performed for this object.When the object does not have a single color, it is ascertained thatJPEG compression should be preformed for the object. Further, when it isdetermined that the image should be treated as a symbol area, the colorsconstituting the area are reduced through a predetermined colorreduction process. When only one color is obtained through the colorreduction process, the palette (e.g., (R, G, B)=(20, 30, 40))representing that color is determined to be an MMR compression target,while being correlated with the binary image. When through the colorreduction process the image can be represented by a predetermined number(e.g., four) of colors or less, each time the symbol cutting process isperformed, palettes representing the individual colors and multi-valuedimages indicating the pixel positions of the colors are correlated witheach other to be determined as ZIP compression targets. When the imagecan not be represented by the predetermined number of colors, theoriginal image before the color reduction process is performed isdetermined to be the JPEG compression target.

FIG. 25 is a diagram showing the configuration used when the presentinvention is employed for the image compression method. An imagebinarizing unit 3102 receives an original image 3101, and optimallybinarizes the original image 3101 to obtain a whole surface binarizedimage 3103. A symbol area detector 3104 receives the complete surfacebinarized image 3103, detects a symbol area, and prepares symbol areacoordinates 3112.

A symbol color extraction unit 3108 receives the symbol area coordinates3112, refers to the original image at the coordinates and the binaryimage 3103 to calculate the original image color in the black portion ofthe binary image, prepares multiple palettes 3114, and performs thecolor reduction process for the original image in accordance with thepalettes 3114.

A symbol portion painting unit 3105 extracts, from the original image,the black portion of the binary image 3103 in an area that is determinedto be a symbol by the symbol area detector 3104 and for which the symbolcolor extraction unit 3108 reduces the number of symbol colors to lessthan M, paints the black portion the color of the surrounding portion,and prepares an image A.

A reduction unit 3106 receives and reduces the image A, and generates animage B.

A JPEG compression unit 3107 receives the image B, and performs JPEGcompression for the image B to generate compressed code X (3113).

A color reduced image 3109 is for multiple symbol areas the colors ofwhich are reduced by the symbol color extraction unit 3108. When thecolor reduced image 3109 is one bit, an MMR compression unit 3110receives the color reduced image 3109 and performs MMR compression toobtain multiple compressed codes Y (3115). For a reduced color image3109 of two bits, a ZIP compression unit 3111 receives this image 3109and compresses it to obtain multiple compressed codes Z (3116). Finally,the data 3112 to 3116 are combined to obtain compressed data 3001A.

Symbol Area Detection Process

FIG. 27 is a flowchart for explaining the processing performed by thesymbol area detector 3104.

At step S3301, a color image is received and brightness conversion isperformed for the color image, while the resolution is reduced bythinning and a brightness image J is obtained. When the original imageis, for example, RGB 24 bites at 300 dpi, the operationY=0.299R+0.587G+0.114Bis performed for every four pixels vertically and horizontally. The newimage J that is obtained is Y8 bits at 75 dpi. At step S3302, thehistogram for the brightness data is prepared, and the binary thresholdvalue T is calculated.

At step S3303, the brightness image J is binarized by using thethreshold value T, and a binary image K is created. Further, at stepS3304, border line tracing is performed for the black pixels, and labelattachment is performed for all the black areas. At step S3305, areasthat are assumed to be symbols are determined in the black area, and atstep S3306 the areas are combined in accordance with their shapes andpositions.

An example of this processing will now be described. A color document inFIG. 4 is received, and a histogram in FIG. 5 is obtained by thinningand performing brightness conversion for the color document. Byreferring to this histogram, the average data and distribution data areemployed to calculate the threshold value T (e.g., 150), and a binaryimage shown in FIG. 6 is obtained. The border line tracing is performedfor the black pixels in FIG. 6, and through label attachment, only agroup of black pixels, the width or the height of which is equal to orsmaller than the threshold value, is identified as a symbol. Then, thegroup of black pixels in FIG. 7 is determined to be a symbol area. Inthis example, the image is shown purely for the sake of the explanationand it is not actually created during the symbol area detection process.

When the black pixels are grouped, depending on whether they arepositioned closely or whether their widths and heights match, 16 symbolareas shown in FIG. 32 can be detected. The coordinate data for thepixels are stored as symbol area coordinates 3112 in FIG. 25.

Instead of binarizing the color image, by using differential filtering,an edge amount relative to an adjacent pixel may be calculated for allthe pixels and binarized, and border line tracing may be performed forthe obtained binary image to detect a symbol area.

Symbol Color Extraction Process for a Symbol Area

FIG. 43 is a flowchart for the processing performed by the symbol colorextraction unit 3108. The complete surface binarized image 103 isemployed in this processing; however, only the coordinates of a symbolarea and a color image may be received, and the image obtained bybinarizing the color image may be employed to perform the representativecolor operation process.

The processing in FIG. 43 is performed for all the areas that the symbolarea detector 3104 determines to be symbol areas.

(Re-binarizing Process)

First, the re-binarization judgement is performed at step S6001.

The complete surface binarized image 3103 is not always the imageobtained by preferably binarizing all the symbol areas. Since thequality of the resultant image is adversely affected, regardless ofwhether the binary image is too thick or too thin, it is ideal for theoptimal binarization to be performed for each symbol area. Since,compared with the complete surface histogram in FIG. 29, a simpler shapeshown in FIG. 33 can be expected for the brightness histogram for eachsymbol area, the threshold value can be easily determined. A portion3901 is a set of background colors, and a portion 3902 is a set ofsymbol colors. In this embodiment, in order to reduce the processingtime, the re-binarization is performed only for a “too thick binaryimage” that has a greater effect on the image quality.

Specifically, the symbol area detector 3104 scans the binary image inthe area that is determined to be a symbol, and performs patternmatching with an isolated point filter. A check is performed todetermine whether the isolated point that is present is equal to orabove the threshold value in the area. When the isolated point is equalto or above the threshold value, the brightness histogram for the areais obtained, the optimal threshold value is calculated, and there-binarization is performed. For a normal symbol area, the brightnesshistogram need only be partially prepared to obtain a better image;however, in some cases, worse results may be obtained (the imageobtained by re-organization may be worse). In order to prevent thisphenomenon, the binarized threshold value that is used to obtain thecomplete surface binary image is entered for the re-binarization, and anexception process is provided in which the re-binarization is notperformed when a binary image is obtained that has a greater densitythan the threshold value for the re-binarization.

(Symbol Cutting Process)

At step S6002, the symbol cutting information is prepared.

The symbol cutting unit changes the process depending on whether thesymbol area is a landscape or a portrait view. The symbol area detectordetermines the landscape or portrait positioning of the symbol portionin accordance with the arrangement of the black blocks, and preparesinformation indicating whether landscape or portrait positioning isused. When landscape positioning is used for the symbol area, first, theblack pixels of the binary image are projected in the main scanningdirection. And when the separation between lines is detected, the blackpixels are projected for each line in the sub-scanning direction, andthe information for each symbol is obtained. When the portraitpositioning is used for the symbol area, the line cutting is performedin the sub-scanning direction, and the symbol cutting is performed inthe main scanning direction. At this time, it is better for the linecutting to be projected to three segments in the line direction in orderto allow the image to be tilted. Through this processing, the coordinateinformation for each line and the coordinate information for the symbolsthat are spaced along each line can be obtained.

In the symbol judgement process (step S6003, which will be describedlater), the symbol cutting information is employed to determine whethereach of black object, in an area that the symbol area detectordetermines to be a symbol, is a symbol. Specifically, whether the blackobject is a symbol is determined in accordance with its size and shape.While taking image quality and data compression into account, it is notnecessarily required that the black object be a symbol in order toconvert it into a single color or a multiple color area (for example,because of a higher image quality and a better compression rate can beobtained, a mark having a single color should be represented by singlecolor MMR rather than the JPEG). However, since it is highly probablethat areas other than the symbol area will be represented by gradation,the determination of the object is required.

(Symbol Judgement Process)

The symbol judgement process is performed at step S6003.

During this process, the symbol cutting information (S6002) is entered,and the average symbol size is calculated for each line. When theinformation for an extremely small symbol is ignored, better results canbe obtained. If the object rectangle is extremely larger than theaverage size, it is determined not to be a symbol, and if the shape ofthe object does not seem from the aspect ratio to be a symbol in spiteof its being of average size, it is also determined not to be a symbol.

When m black objects are present in the area, and when all the m blackobjects are determined not to be symbols, the symbol judgement unitoutputs a determination that the area is an image.

When n black objects (m>n, n≧0) among m black objects are not symbols,i.e., when rectangles that do not indicate symbols remain, a blackobject on the binary image that is determined not to be a symbol isdeleted, and a determination that the pertinent area is a symbol isoutput.

The following exception process is added while taking into account thefinal image quality. When five symbols out of ten in an area arerepresented as single color symbols, and when the other five symbols areregarded as not being symbols and JPEG compression is performed forthem, an uneven image is obtained, which visually is not preferable.Thus, for a case wherein the symbol judgement unit frequently changes adetermination for either a symbol or an image, in accordance with thearrangement and the frequency at which the rectangle is determined to bea symbol, all the objects are determined to be rectangular symbols, orrectangular images.

When an area is determined to be a symbol during the symbol judgementprocess, program control advances to step S6004. And when the area isdetermined not to be a symbol, program control is shifted to step S6005.

(Monochrome Judgement)

The monochrome judgement process is performed at step S6005.

An area to be processed here is an area that is determined by the symbolarea detector to be a symbol, but is determined not to be a symbolduring the symbol judgement process. As is described above, regardlessof whether the area is a symbol, it is better that the monochromeprocess is performed for the area represented by a single color and thatMMR compression be performed for the results, so that a higher imagequality and a higher compression ratio are obtained. Thus, a process isperformed to determine whether the area is or is not monochrome.

As a specific example, histograms are obtained for the RGB levels of thepixels of a color image that correspond to black portions of the binaryimage, and when all the distribution values of the histograms are equalto or greater than the threshold value, it is ascertained that the areais monochrome.

When the area is monochrome, program control advances to step S6006 fora one color extraction, and when the area is represented by multiplecolors, the processing is terminated.

(One Color Extraction Process)

The one color extraction process at step S6006 will now be describedwhile referring to the flowchart in FIG. 36.

At step 4202, the thinning process is performed for a binary image newbithat is referred to by symbol coordinates, while the number of blackobjects in a portion wherein the color is shifted from the background toa symbol portion during scanning is reduced, and a new binary image iscreated. At step S4203, the histogram is obtained for the RGB values ofthe original image that corresponds to the black pixels of the imagenewbi (the histogram for another color space, such as a YUV space, maybe prepared). At step S4204, the representative values for RGB areobtained, and in this case, the greatest values may be employed. Or,another method may be employed whereby the greatest value is obtained byusing a rough histogram that is prepared using a reduced number ofsteps, and detailed histograms that are present in the rough histogramare employed to obtain the greatest value.

Using this method, a true representative value 4301 can be obtained fromthe histogram shown in FIG. 37 without it being disturbed by noise 4302.A detailed explanation will be given while referring to FIG. 37. Adetailed histogram of 256 levels in FIG. 37 can be obtained, forexample, from R data of eight bits. Since the maximum value is 1302,which is not a true representative value, the histogram is divided by 64into eight segments that overlap each other, and the eight segments arere-calculated using the histogram for the 256 levels. The obtainedsegments 0 to 8 are shown; however, segments 0 and 8 each have only 32levels. It is found through recalculation that a representative value ispresent in segment 6, and segment 6 is searched to obtain the maximumvalue 4301. The above processing is repeated for all the symbolcoordinates, and one representative color is calculated for each of thesymbol coordinates.

(Color Reduction Process)

At step S6004, the color reduction process is performed for the symbol.

In the processing performed by the color reduction unit 4082, even whenthe original document is represented by a single color, the portionwherein the color is shifted from the background to the symbol portionis present during the scanning.

FIGS. 46 and 49 are diagrams showing the color shifting caused byscanning. In FIG. 46, to simplify the explanation, instead of RGB, onlyR is employed. Symbol A is originally represented by a single color atlevel R=(32, 32, 32); however, when the symbol A is read by the scanner,the data for this symbol are distributed, as is indicated by enlargedpixels. There are only three pixels 6201, 6202 and 203 that reach theblack color that is close to the original level R=(32, 32, 32), and theother pixels are positioned between the background color (white in thiscase) and the level R=(32, 32, 32), so that the symbol is represented bya gradation that is a shifting portion. FIG. 49 is a diagram showing thestate wherein the color is shifted by using the three-dimensionalhistogram of the pixel level of the symbol A in FIG. 46. Assume that thebackground color is white 6501, and the symbol color is black 6502, anda portion 5603 is the shifting portion.

It is not necessary to strictly represent the shifting portion thatconstitutes a variance due to the scanning of the symbol portion that isoriginally represented by a single color. When the shifting portion canbe represented using only the representative color, a high image qualityis obtained, and the amount of data required is reduced. However, evenwhen the thinning process is performed for a binary image, it isdifficult for the color of the shifting portion in the symbol area to becompletely removed from the background. Therefore, by using the factthat one symbol tends to be represented by a single color, the symbolcutting information is employed to limit the colors to one per symbol,so that the improvement of the image quality and the compression rate isthe objective. It should be noted that, when a symbol originallyrepresented by gradation is to be compressed at a high image quality,only one additional exception process, such as a determination as towhether the symbol is represented by multiple colors, need be performed.That is, when one symbol is limited to one color by using the symbolcutting information, it is possible to remove the shifting portion thatis generated as a variance due to the scanning of a symbol image thatoriginally is represented by a single color.

The processing performed by the color reduction unit will now bedescribed in detail while referring to the flowchart in FIG. 44.

At step S6102, the thinning process is performed for a binary image thatis referred to by the symbol coordinates, and the number of blackportions that correspond to the shifting portions whereat the color isshifted from the background to the symbol portion during scanning isreduced, and a new image “thinimage” is prepared. Since the binary image“thinimage” is used for the process at step S6110, this image isconstituted by eight bytes having binary values of 255 (black) and 0(white). At step S6103, the three-dimensional histogram is obtained forthe RGB colors of the original image that corresponds to the black pixelof “thinimage”. At this time, when, for example, the input image has RGBcolors of eight bits each, a 256*256*256 histogram is normally required.While taking into account that it is not the gray level but theresolution that is necessary for the symbol portion, and that a littledifference in the pixel values should be ignored in order to calculatethe representative color while the variance is reduced during thereading performed by the scanner, the histograms at many such levels isnot actually required. Therefore, in this example, a RGBthree-dimensional histogram of the upper five bits is obtained. Toobtain the histogram, the total number blacknum of black pixels that arepresent in the symbol area is also calculated.

In this embodiment, the RGB space is employed; however, another colorspace such as Lab or YUV may also be employed. Further, athree-dimensional histogram is employed; however, three one-dimensionalhistograms may be employed for the individual colors.

At step S6104, the initial process is performed in which the numbercolnum of the symbol colors that are represented in the area is reset,or in which the number okpixel of the processed pixels is reset. And atstep S6105 the representative value is calculated. In this case, thepoint whereat the total value of the seven histograms, including thetarget histogram, reaches the maximum value is employed as arepresentative value (seven histograms: a target point, two adjacentpoints in the R dimension, two adjacent points in the G dimension andtwo adjacent points in the B dimension (see FIG. 39)). The thus obtainedmaximum value is substituted into color[colnum], colG[colnum], andcolB[colnum].

The range, with the representative value as the center, of the color tobe converted into the representative value is determined.

The representative values are fixed to obtain three one-dimensionalhistograms. FIG. 47 is a diagram showing the three one-dimensionalhistograms that are obtained. For example, when the representative valueis (Color(26), ColG(30), ColB(22), the R one-dimensional histogram (=allthe histograms are projected along a line 6301) wherein thethree-dimensional histogram G and B are fixed at 30 and 22, the Gone-dimensional histogram (=all the histograms are projected along aline 6302) wherein the three-dimensional histogram R and B are fixed at26 and 22, and the B one-dimensional histogram (=all the histograms areprojected along a line 6303) wherein the three-dimensional histogram Rand G are fixed at 26 and 30 are obtained. For example, the Rone-dimensional histogram has the shape shown in FIG. 48, and points6401 and 6402 are detected therefrom, and the “R range” is definedwherein these points are used as the representative values. The methodfor determining a binarized threshold value for an image is employed forthe detection of the points 6401 and 6402. While, for example, a point6403 is a representative value, the histogram including 0 to therepresentative value is substituted into the binarized threshold valuedetermination function to obtain the point 6401, and the invertedhistogram of the histogram including the representative value at stepS31 in FIG. 48 is substituted into the binarized threshold valuefunction to obtain the point 6402.

The color range is determined for R, G and B, and is substituted intofg_range[colnum].

At step 6106, all the values of the three dimensional histogram infg_range[colnum] are set to 0. At this time, the number of pixels thatare set to 0 is added to the okpixel that represents the number ofprocessed pixels.

At step S6107 the approximation color judgement is performed. Thisprocess is performed for all the colors (from fg_color[0] tofg_color[colnum·1]) that have appeared. When the approximation color isfound, the processing loop is exited. As was explained for the symbolcutting, in the image obtained by the scanner, the gradient color occursbetween the background color and the symbol color. In other words, as isshown in FIG. 49, a color pixel (6503) is present between the backgroundcolor (6501) and the symbol color (6502). By referring to the binaryimage, the color from the line 6504 that is nearer the background coloris not added to the three-dimensional histogram (the line 6504 ispositioned nearer the symbol color side by thinning the binary image).However, the gradient portion is still present, and the color of thisportion would be extracted after the symbol color 6502 has beenextracted. For example, while fg_color[0] is (32, 40, 40), (96, 112, 96)tends to be extracted for fg_color[m]. The approximation color judgementis performed in order to determine that these two are the same color.Since this judgement is a little difficult in RGB space that is not auniform color space, the judgement is performed in the Lab space. WhenLab conversion is performed for fg_color[0]=(32, 40, 40), (15, −4, −1)is obtained, and when the Lab conversion is performed forfg_color[m]=(96, 112, 96), (45, −9, −7) is obtained.

Through this processing, it is possible to remove the gradient portionthat has occurred as a variance when the scanner reads a symbol imagethat originally is represented by a single color, and high quality imagecompression can be performed at a high compression rate.

Actually, the background color should also be detected, and the colorthat is present along the extended line from the background color andfg_color[0] should be determined to be the approximation color. However,in this embodiment, it is assumed that many white portions are includedin the background, and when a distance of “ab” in the Lab space is equalto or smaller than the threshold value, the pertinent color isdetermined to be the approximation color. The determination result ismaintained in the kinji[ ] matrix. The colors having the same numbers inkinji[ ] represent the approximation color.

An example is shown below, where kinji[0] and kinji[3] are both 0. Thatis, fg_color[0] and fg_color[3] are determined to be approximationcolors.

fg_color[0]=(32, 40, 40) kinji[0]=0

fg_color[1]=(248, 64, 48) kinji[1]=1

fg_color[2]=(48, 256, 32) kinji[2]=2

fg_color[3]=(96, 112, 96) kinji[3]=0

At step S6018 the colnum is incremented.

At step S6019 a check is performed to determine whether the number ofblack pixels for which color extraction has been completed has exceeded75% (“75” is merely an example).

That is, a check is performed to determine whetherokpixel*75>blacknum*100 is satisfied.

If the number of black pixels has exceeded 75%, program control advancesto step S6110. If the number of black pixels has not exceeded 75%,program control is shifted to step S6113.

At step S6110, a palette image is formed in the “thinimage”.Specifically, the pixel RGB level of the color image wherein the valueof the “thinimage” corresponds to pixel 255 (indicating color allocationhas not yet been performed) is referred to, and when the RGB data ispresent in fg_range[m], the value of kinji[m]+1 (i.e., a value of 1) issubstituted into the pixel value that “thinimage” corresponds to. Inthis case, instead of kinji[m], kinji[m]+1 is substituted because sincea value of 0 is a special number representing a non-symbol portion(background portion), when kinji[m] is 0 it can not be substituted.

When, at step S6110, the palette image is formed in “thinimage”, at stepS6111 color information charpal for each symbol is prepared from thepalette image by using the symbol cutting information.

A method for preparing the color information charpal for each symbol cutunit will now be described by using a symbol image in FIG. 50 as anexample. In the symbol portion “

.

.”, it is assumed that “

” represents the red extracted by fg_color[1], “

” is the blue detected by fg_color[2], and the other symbols are black.

In the symbol cutting process, “

” at the beginning of the symbols are processed as the first symbol.Assume that the number of black pixels in the thin-line image“thinimage” present in this symbol is 100, and that 70 pixels out of 100are present in fg_range[0] and 20 are present in fg_range[3]. Sincekinji[0] and kinji[3] are both 0 (approximation colors), a value of 1 isallocated to 90 pixels at step S6110. At this time, when one numberoccupies the maximum number of existing black pixels, that color isallocated. In this example, ten pixels remain to which color has not yetbeen allocated, and even if the colors of these pixels are detected, thetotal number of colors will not exceed 90. Thus, the color informationcharpal[0] for “

” is determined to be 1.

In this manner, the color allocated to each symbol cut potion isselected from among three charpals, 0 to 2. As a result, as the colorinformation charpal, kinji[1]+1=2 indicates that red is allocated forthe left radical of “

” and the right radicals “

” of “

” and “

”; kinji[2]+1=3 indicates that blue is allocated for “

”; and kinji[0]+1=1 indicates that black is allocated for the remainingsymbols.

When different values uniformly appear in multiple kinji[ ], a symboltends to be represented by multiple colors. However, this determinationis limited to those occasions when the approximation color judgementprocess functions preferably. Thus, when this function is included,performance of the approximation color judgement process in the Labspace is recommended.

In this example, since the color information charpal is allocated forall the existing twenty symbols, this state matches the condition atstep S6113 for exiting the symbol color extraction loop and programcontrol jumps to step S6114. When, however, the color information isallocated to only 18 symbols out of 20, at step S6112, thethree-dimensional histogram is again obtained only for those symbols forwhich charpal was not prepared, and program control returns to stepS6105. At this time, the number of black pixels of a thin symbol forwhich charpal was is not prepared is substituted into blacknum, and“okpixel” is reset. Then, in the above example, since symbol colorextraction is completed up to fg_color[3], beginning with fg_color[4]the extraction is performed.

At this time, to increase the processing speed, re-acquisition of thethree-dimensional histogram at step S6112 may be performed only onetime, and the processes at step S6110 and S6111 may be limited to tworepetitions. Experiments have shown that is enough to obtain asatisfactory image quality.

Under the above limitations, when program control exits the loop at stepS6113, there could still be a symbol for which charpal has not beenprocessed. Therefore, when the number of symbols nokorichar for whichcharpal has not been processed is equal to or greater than one, at stepS6114, the color information is forcibly allocated to the unprocessedcharpal. Specifically, in the allocation process at step S6111, onlywhen the maximum value is obtained is the color information substitutedinto charpal, while taking into account the succeeding trend of thepixel 255 of “thinimage” (i.e., the pixel for which color extraction hasnot yet been performed). However, at step S6114, the pixel 255 of“thinimage” is ignored, and the maximum value among the pixels otherthan 0 (background) is used to determine the value for charpal. Since isa case exists wherein the pixels other than 0 (background) are all 255(color extraction is not performed even for one pixel), the color ofcharpal for a symbol positioned nearby is substituted into charpal. Forexample, when “

” in FIG. 50 corresponds to the conditions in this case, charpal of thenearby positioned “

” is substituted in.

At this time, the numerical value “colnum” is obtained as the number ofcolors that have been extracted. However, since this value includes theapproximation color and also may include a color that is not used, eventhough it was extracted, the value colnum differs from the number ofcolors actually used (usecolnum). Thus, at step S6115, the charpal isexamined to calculate the number of colors that are actually used.

When, at step S6116, “usecolnum” calculated at step S6115 is equal to orgreater than 16, 17 colors including the background (0) are present andcan not be represented by four bits. Thus, program control advances tostep S6117, the color reduction process for this area is abandoned, andDOJPEG is returned (the area is represented as the background image). Itshould be noted that when the use of eight bits rather than four bits ispermitted, the usecolnum is not 16 but 256.

When usecolnum is one, program control is shifted to step S6118, and apalette of one color is prepared for use. At step S6119, the inputbinary image is clipped, and at step S6120 DOMMR is returned.

When usecolnum is equal to or greater than two and smaller than 16,program control is shifted to step S6121, the palette of colors to beused is prepared, and at step S6122, a palette image is created. In thisexample, when usecolnum is two or three, a palette represented as usingtwo bits for each pixel is created, while when the usecolnum is equal toor greater than four and smaller than 16, a palette represented as usingfour bits for each pixel is created. At step S6123, DOZIP is returned.

When the thus obtained color reduced image 3109 has one bit (DOMMR isreturned as the result of symbol color extraction), the MMR compressionunit 3110 performs MMR compression for the image 3109 and prepares thecompressed code Y. When the color reduced image 3109 has two or morebits (DOZIP is returned as the result of symbol color extraction), theZIP compression unit 3111 compresses the image 3109 and prepares thecompressed code Z. When DOJPEG is returned, the image 3109 is nottransmitted to the MMR compression unit 3110 or the ZIP compression unit3111, whereat the reduced color image is not present, and a command istransmitted to the symbol portion painting unit 3105 so as not to treatthe image as a symbol area.

Symbol Painting Process

The processing performed by the symbol portion painting unit 3104 willnow be described while referring to FIGS. 34A to 34C and 35. FIG. 35 isa flowchart showing the symbol portion painting processing.

As an example, assume that the image shown in FIG. 34A, wherein agradation image is used as a background and blue symbols ABC are drawnsubstantially in the center, is employed as an original image, and thatthe binary image of one symbol area shown in FIG. 34B is obtained fromthe original image. In the symbol portion painting process, first, atstep S4101 the entire image is divided into 32×32 areas (hereinafterreferred to as parts), and the process is performed for each part. Thestate obtained during this process is shown in FIG. 34C. To simplify theexplanation, 5×5 parts are shown, and the numbers in the upper leftportions of the parts indicate the part numbers. The number of parts isnot limited to the number used here, and an image may be divided into adifferent number of parts.

At step S4102, a check is performed to determine whether a part has notbeen processed. When a part has not been processed, at step S4113, acheck is performed to determine whether the target area for the symbolportion painting is present in that part. An area for which the symbolcolor extraction unit 3108 has returned the DOJPEG is not regarded as asymbol portion painting target, even though that area is determined bythe symbol area detector 3104 to be a symbol area.

In the example in FIG. 34C, it is ascertained at step S4103 that thereare no symbol portion painting target areas in parts 00 to 04, 10, 14,20, 24 and 30 to 35, and without performing any process for them, thenext part is processed. For the part (e.g., part 1) wherein a symbolportion painting target area is present, at step S4104, by referring toa corresponding binary image, the average value ave_color is calculatedfor the RGB values (or YUV values) of the color image that correspondsto the white portion of the binary image. Then, at step S4105, thecorresponding binary image is referred to, and the pixel density datafor corresponding black pixels are determined to be ave_color. The abovedescribed processing is repeated for parts (parts 12, 13, 21, 22 and 23)wherein symbol painting target areas are present. As a result, theaverage value of peripheral pixels can be embedded in the portionswherein the symbol is present.

The obtained image is reduced by the reduction unit 3106. In thisembodiment, a simple thinning-out process is employed for sizereduction. The reduction process and the symbol portion painting processmay invertedly be performed. In this case, position shifting between thebinary image and the color image must be taken into account.

Further, if necessary, a format is prepared by collecting the symbolarea coordinates 3112, the palette 3114, the compressed code X 3113, thecompressed code Y 3115 and the compressed code Z 3116. An example formatfor collecting these five is the PDF of Adobe (trademark). The PDF ofAdobe is a format that is displayed by the application “Acrobat Reader(trademark)”, which is distributed by Adobe for free, and a problem suchas one where a reception side that does not have an application toprepare a document, and thus can not open a file, can be avoided.Another example format is XML. XML is a descriptive language used forthe exchange or distribution of documents or data via a network.

Expansion Process

FIG. 26 is a diagram showing the arrangement required for the expansionprocess.

A JPEG expansion unit 3201 performs JPEG explanation for the receivedcompressed code X 3113, and prepares a multi-valued image E. Anenlargement unit 3202 receives the multi-valued image E, and enlarges itto obtain a multi-valued image F 3203. An MMR expansion unit 3204receives the compressed code Y 3115, and prepares a binary image G 3205.An IP expansion unit 3206 receives the compressed code Z 3116, andcreates a multi-color image H 3207. And an image combining unit 3208receives the symbol area coordinates 3112, and corresponding palette3114 and binary image G 3205 or multi-color image H 3207; selects thecolor of the pixel of the image F 3203 when the image data of the binaryimage or the multi-color image represents transparency, or selects acorresponding palette color in the other case; and creates a final imageI 3209.

FIGS. 38A to 38C are diagrams showing the results obtained by thecombining unit 3208. First, the compressed code C obtained by JPEGexpansion is shown in FIG. 38A. For this process, the image in FIG. 34is employed; however, when the quantization irreversible method for JPEGcompression is employed, data are obtained whose pixel value slightlydiffers from that in FIG. 34C. Compared with a case wherein the originalimage before the symbol portion is extracted is compressed by JPEGirreversible compression, a change in the pixel value is small when thesame quantization table is employed. In other words, a high qualityimage is obtained. In this embodiment, the binary image obtained by MMRcompression is used as the symbol area for which the combining processis to be performed. The binary image obtained by the expansion is shownin FIG. 38B. Assume that the palette for this image is R=20, G=30 andB=225. By referring to the binary image in FIG. 38B, the palette color(20, 30, 255) data are placed on the image portions in FIG. 38A thatcorrespond to black pixels, and finally, the image shown in FIG. 38C isobtained. For a multi-color image, the number of palettes is changed.For example, for two bits, the palettes allocated for four pixel values00, 01, 10 and 11 are applied. One of the pixels representstransparency, and when, for example, it is 00, the pixel in FIG. 38A isselected for this pixel.

For the pixel value of 01, the palette value of 01 is applied; for thepixel value of 10, the palette value of 10 is applied; and for the pixelvalue of 11, the palette value of 11 is placed. As a result, theexpanded image 3209 is obtained.

<Modification>

In the above embodiments, the binary image is created by using thecomplete surface single threshold value. However, another thresholdvalue may be employed; for example, an optimal threshold value may becalculated for each symbol area detected by the symbol area detector3104, and a binary image may be created. In this case, there-binarization determination process at step S4001 in FIG. 43 is notrequired.

Further, the same binary image has been used by the symbol portionpainting unit 3105 and the symbol color extraction unit 3108; however,the same binary image need not always be used, and an optimal binarizingunit may be internally provided for the units 3105 and 3108.

In addition, in the above embodiments, while it is impossible to processa symbol (inverted symbol) having high brightness on a background havinglow brightness, this process can be performed by using, for example, thearrangement in FIG. 40. A differential processing unit 4702 performs thedifferential filtering shown in FIGS. 41A and 41B for pixels with atarget pixel as the center, and binarizes the pixels in such a mannerthat, when the absolute value of the pixel value exceeds the thresholdvalue, the pixel is determined to be black, and when the absolute valuedoes not exceed the threshold value, the pixel is determined to bewhite. The primary differential filter is shown in FIG. 41A. Thehorizontal line can be detected by the upper portion, while the verticalline can be detected by the lower portion, and the oblique line can bedetected by using the total of the absolute values of two filters.Further, the oblique line filter may also be employed. A secondarydifferential filter that copes with all the directions is shown in FIG.41B. The secondary differential filter can also be prepared forhorizontal detection and vertical detection. This filter is positionedfor all the pixels, and a differential image 4702 is created. At thistime, when the filtering is performed while thinning the pixels, theresolution can be reduced at the same time. When the process beginningat step S3303 in FIG. 27 is performed for the thus obtained binaryimage, the coordinates of a symbol area that includes an inverted symbolcan also be detected. To also extract the inverted symbol as a symbolarea, the binarizing unit 4703 must cope with this process. When theinverted symbol area is extracted as a symbol area, mainly the patternin FIG. 42 is received, while it has been assumed in the aboveembodiments that only the pattern in FIG. 33 is received. The case inFIG. 42B is for the inverted symbol, and the case in FIG. 42C is wheresymbols of two colors, a black symbol and a white symbol, are present inthe same gray background. While taking these three patterns intoconsideration, the binarizing unit 4703 need only detect points A and Band perform the binarization process, so that the area sandwichedbetween the points A and B is white and the other area is black. Or,disregarding the case in FIG. 42C, only one threshold value along whichthe background and the symbol portion are separated need be detected,and for an inverted pattern, the inversion pattern may be performed.When the inverted symbol area can be coped with in this manner, theinverted symbol area, which remains in the JPEG compressed image in thefourth embodiment, can be smoothed through the symbol potion paintingprocess. As a result, compression efficiency is improved, and theinverted symbol portion can be compressed without a reduction in theresolution or deterioration due to mosquito noise.

The present invention may be employed for a system that is constitutedby multiple apparatuses (e.g., a host computer, an interface device, areader or a printer), or for one apparatus (e.g., a copier or afacsimile machine).

Further, the objective of the invention can also be achieved bysupplying, to a system or an apparatus (or a CPU or an MPU), a storagemedium (or a recording medium) on which software program code thatimplements the functions of the embodiments is recorded, and bypermitting the system or the apparatus to read and execute the recordedprogram code. In this case, the program code read from the storagemedium provides the functions of the above described embodiments, andthe storage medium on which the program code is recorded constitutes thepresent invention. In addition, with the present invention it is notonly possible for the functions of the previous embodiments to beprovided by the execution of program code by the computer, but also, theprogram code can interact with an Operating System (OS) running on acomputer, or with another software application, to provide the functionsdescribed in the above embodiments.

Furthermore, with the present invention, program code, read from astorage medium, can be written in a memory that is mounted on a functionexpansion board inserted into a computer, or in a function expansionunit connected to the computer, and in consonance with instructions inthe program code, a CPU mounted on the function expansion board, or inthe function expansion unit, can perform part or all of the actualprocessing required to implement the functions of the above describedembodiments.

When the present invention is applied for the above storage medium, theprogram codes that correspond to the above described flowcharts (FIG.43, and/or FIGS. 44 and 45) are stored on the storage medium.

According to the present invention, since one color is allocated foreach symbol cut unit, the symbol can be efficiently represented bymultiple colors, and when this method is used for the compressionsystem, high quality image compression can be performed at a highcompression rate.

1. An image processing apparatus comprising: histogram calculation meansfor calculating a histogram that is consonant with an input image;binary threshold value calculation means for calculating a binarythreshold value, based on the histogram, with which a predetermined areain the input image is blurred; binarization means for binarizing theinput image using the binary threshold value; and calculation means forcalculating the color of the predetermined area of the input image basedon the results obtained by said binarization means.
 2. An imageprocessing apparatus according to claim 1, wherein, by referring to abinary image obtained by said binarizing means, said calculation meanscalculates the average value of portions in the input image thatcorrespond to black portions of the binary image, and calculates thecolor of a predetermined area of the input image based on the averagevalue.
 3. An image processing apparatus according to claim 1, wherein,by referring to a binary image obtained by said binarizing means, saidcalculation means calculates a histogram for portions in the input imagethat correspond to black portions of the binary image, and calculatesthe color of a predetermined area of the input image based on thehistogram.
 4. An image processing apparatus according to claim 1,wherein said binarizing means further includes inversion means forinverting the binarized results.
 5. An image processing apparatusaccording to claim 1, wherein the image in the predetermined area is asymbol image.
 6. An image processing method comprising: a histogramcalculation step of calculating a histogram that is consonant with aninput image; a binary threshold value calculation step of calculating abinary threshold value, based on the histogram, with which apredetermined area in the input image is blurred; a binarization step ofbinarizing the input image using the binary threshold value; and acalculation step of calculating the color of the predetermined area ofthe input image based on the results obtained at said binarization step.7. An image processing method according to claim 6, wherein, byreferring to a binary image obtained at said binarizing step, at saidcalculation step, the average value of portions in the input image thatcorrespond to black portions of the binary image is calculated, and thecolor of a predetermined area of the input image is calculated based onthe average value.
 8. An image processing method according to claim 6,wherein, by referring to a binary image obtained at said binarizingstep, at said calculation step, a histogram for portions in the inputimage that correspond to black portions of the binary image iscalculated, and the color of a predetermined area of the input image iscalculated based on the histogram.
 9. An image processing methodaccording to claim 6, wherein said binarizing step further includes aninversion step of inverting the binarized results.
 10. An imageprocessing method according to claim 6, wherein the image in thepredetermined area is a symbol image.
 11. A computer-readable storagemedium on which stored is a program comprising: a code for calculating ahistogram that is consonant with an input image; a code for calculatinga binary threshold value, based on the histogram, with which apredetermined area in the input image is blurred; a code for binarizingthe input image using the binary threshold value; and a code forcalculating the color of the predetermined area of the input image basedon the results obtained at said binarization.